Conversation
…on memo
resolve_runtime_provider() calls load_config() (a full defensive deepcopy,
~265us warm) 5-7x per call via the resolve tree — measured 74% of the
~2.35 ms/call cost — and the gateway resolver calls it 5-8x per inbound
message. Two layers fix this:
Layer A (hermes_cli/runtime_provider.py): every config read in the resolve
tree is read-only, so switch all 6 sites to load_config_readonly()
(mtime-cached, no deepcopy). _get_model_config() deep-copies only the small
model section to preserve its mutation-safety contract. Measured:
resolve_runtime_provider 2.35ms -> 0.94ms/call; _get_model_config 0.38ms ->
0.023ms/call.
Layer B (gateway/run.py): the MoA path already caches
resolve_runtime_provider behind a 300s TTL (merged MoA runtime cache,
agent/moa_loop.py); the gateway path now gets the same treatment via
_memoized_resolve_runtime, keyed on the files the resolution reads
(config.yaml + profile/global auth.json) with a TTL backstop for
env-var-only changes. Vertex (per-call OAuth token) and AuthError/fallback
results are never cached. The memo stores a copy and hands out fresh dicts
so caller mutation (pop('model')) can never corrupt it. Per-message:
~12-19ms -> ~1.5-2ms (first call resolves, the rest hit the memo).
Tests: new tests/gateway/test_runtime_resolve_fastpath.py pins the
measured-work (no deepcopy from the resolve tree, one resolve per message),
TTL/config/auth invalidation, vertex bypass, and mutation safety. Existing
tests that patched rp.load_config as their config-injection seam now patch
rp.load_config_readonly (the seam moved with the loader).
The per-message memo in _memoized_resolve_runtime() keyed on config/auth file (mtime_ns, size) signatures only. A multiplex gateway resolves multiple profiles' agents in the same OS process (the desktop tui_gateway switches profiles per request via set_hermes_home_override), and 'hermes profile create --clone-all' copies the profile tree with mtime-preserving shutil.copy2 — so a cloned profile's config.yaml/auth.json carry the source's identical (mtime_ns, size) and compute the same memo signature, letting one profile receive the other's cached api_key/base_url for up to the 300s TTL. Add str(get_hermes_home()) to the memo key, the same profile-boundary fix open PR NousResearch#78185 applies to agent/moa_loop.py's sibling _runtime_cache. Tests: new tests/gateway/test_runtime_resolve_fastpath_profile_isolation.py drives _memoized_resolve_runtime under two profile overrides and asserts each profile gets its own credentials; both tests fail without the fix and pass with it. Same-profile memo hit preserved (2 invocations, 1 resolve).
|
this is the same profile-boundary gap #78185 closes for What changed: Verification:
|
|
Thanks @spfcraze for this. I reviewed it against current hermes_cli/config.py::load_config_readonly exists on main (544c31b) and load_config is signature-cached; gateway/run_agent_cache.py caches the per-session agent so resolve_runtime_provider is no longer called 5-8x per inbound message (callers: run_agent_cache.py L168 on cache build, run_turn.py L244 on channel override). hermes_cli/runtime_provider.py still has 4 load_config() (deepcopy) reads in the resolve tree. Landed on main in 544c31b perf(agent-loop): cut 47% of per-conversation function calls via 3 targeted hot-path optimizations (#28866). Closing as superseded. Thanks for flagging the hot path. If something here is still slow on a current build, please open an issue with a measurement and we'll look again. |
What does this PR do?
resolve_runtime_provider() deep-copies the whole config 5-7x per call (measured 74% of its 2.35ms/call cost) and the gateway resolver calls it 5-8x per inbound message (~12-19ms of CPU before every LLM call). The MoA path already caches it behind a 300s TTL (#66793); the gateway path never got that. Fix: (A) convert all 6 read-only config reads in the resolve tree to load_config_readonly() (mtime-cached, no deepcopy), deep-copying only the small model section in _get_model_config to preserve its mutation contract — measured 2.35ms->0.94ms/call; (B) add _memoized_resolve_runtime at the gateway per-message entry, keyed on config+auth mtimes with a 300s TTL backstop, never caching the vertex OAuth path or fallback results — measured 0.94ms->0.149ms on memo hit. Per-message cost drops ~10x. 12 new measured-work/behavior-parity tests; sabotage PASS.
Related Issue
No GitHub issue — discovered via code review and reproduced live (see below). Happy to file one first if preferred.
Changes Made
fix/provider-runtime-fastpath— 9 file(s) changed vs base:gateway/run.pyhermes_cli/runtime_provider.pytests/agent/test_bedrock_adapter.pytests/agent/test_nous_portal_anthropic_wire.pytests/gateway/test_runtime_resolve_fastpath.pytests/hermes_cli/test_canonical_custom_identity.pytests/hermes_cli/test_custom_provider_identity.pytests/hermes_cli/test_runtime_provider_resolution.pytests/tui_gateway/test_custom_provider_session_persistence.pyHow to Test
Validation completed:
tests/gateway/test_runtime_resolve_fastpath.py.<full suite>: branch 27401 passed / 6 failed vs baseline 27387 passed / 8 failed — zero branch-only failures.Logs
Sabotage verification output: